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Improved Dark Frame Subtraction 
Background Information 

Field of the Invention 

This invention is generally related to electronic imaging 
systems and more particularly to techniques for correcting the response of 
image sensors. 

Description of Related Art 

Electronic imaging is becoming increasingly popular as 
imaging systems such as the digital camera and personal computer (PC) 
provide consumers with the ability to capture digital images and movies in 
a low cost manner and display and communicate them to others using a PC 
and a network. Another part of the appeal of electronic imaging lies in its 
ability to electronically correct for imperfections in a captured image from 
an expected standard. These imperfections are often caused by the realities 
of manufacturing an imaging system which is to conform with an expected 
specification. 

Non-idealities in the imaging system can cause deviations in 
the response of the system. To correct such deviations, a conventional 
calibration procedure may be performed to electronically adjust the system 
response to match the expected response. 

In one such technique, the imaging system is exposed to 
known color stimuli, such as a Macbeth color chart, to produce picture 
frames (containing images) of the stimuli. Each picture frame comprises a 
number of digital pixel values each one being associated with a sensor 
circuit element in a semiconductor image sensor. The picture frames are 
then compared to an expected specification, and calibration factors related to 
differences between the actual and expected values are computed. The 
calibration factors are then applied to subsequent picture frames taken using 
the same image sensor to result in images having more accurate color 
content. 



WO 99/62023 PCT7US99/1 0699 

2 

In addition to manufacturing process variations, thermal 
processes and defects within the semiconductor structures that make up the 
image sensor can also cause subtle deviations in the response of an imaging 
system. These deviations are referred to herein as dark offset noise. 
Contributors to the dark offset noise are leakage currents (dark currents) and 
circuit offsets in each sensor circuit element of the image sensor. The dark 
offset noise permeates each picture frame. 

The leakage currents and hence the dark offset noise can be 
represented by a dark frame separate from the picture frame captured by the 
image sensor. Most imaging systems such as digital cameras attempt to 
cancel the dark offset noise, to a first order, by simply subtracting the dark 
frame from the picture frame to obtain a corrected frame. The picture frame 
is an image of the desired scene, and the dark frame is another image 
obtained with the same camera but with the camera's mechanical shutter 
being closed so that no light is incident on the image sensor. The 
subtraction improves the quality of the image in the corrected frame by 
canceling, to a limited order, the dark offset noise from the picture frame. 
The noise would otherwise result in speckles or graininess appearing in the 
uncorrected picture frame. 

The conventional dark current subtraction technique described 
above may be combined with conventional calibration techniques to 
achieve somewhat improved accuracy in calibration. However, such a 
technique fails to provide sufficiently accurate and repeatable results from a 
measurement capability analysis of calibration schemes used with 
complimentary metal oxide semiconductor (CMOS) image sensors. It is 
desirable, therefore, to further improve the accuracy and repeatability of 
calibration for image sensors. 

Summary 

The invention in one embodiment is directed at a method of 
adjusting at least one portion of a dark frame in accordance with 
compensation values related to dark reference pixels of a picture frame. 
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The dark frame and the picture frame were obtained from the same image 
sensor. This yields an adjusted dark frame portion which is then subtracted 
from a corresponding portion of the picture frame to obtain a corrected 
picture frame. 

Brief Description of the Drawing s 

Figure 1 presents a framework for an embodiment of the invention. 

Figure 2 illustrates a series of operations to be performed in the 
framework of Figure 1. 

Figure 3 illustrates a technique of obtaining an adjusted dark frame 
according to a another embodiment of the invention. 

Figure 4 illustrates a technique for obtaining an adjusted dark frame 
according to yet another embodiment of the invention. 

Figure 5 illustrates a technique for obtaining adjusted movie frames 
according to an embodiment of the invention. 

Figure 6 illustrates a block diagram of an image processing system as 
another embodiment of the invention. 

Detailed Description 

As briefly summarized above, the invention in one 
embodiment is a method for improving the accuracy of manufacturing test 
and calibration, as well as improving the performance of image sensors and 
imaging systems, by subtracting a better estimate of an ideal dark frame 
from the picture frame. The ideal dark frame is a hypothetical frame that 
would be captured at the same time as the picture frame. In practice, 
however, the dark and picture frames are captured at different times. Even 
a one second time difference between the frames causes the temperature of 
the image sensor to change which in turn causes the dark frame to contain 
substantial errors with respect to the ideal dark frame. The temperature 
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change is particularly strong in sensors having integrated functions, such as 
CMOS image sensors. 

To obtain a more accurate estimate of the ideal dark frame, the 
dark frame is adjusted based on compensation values related to dark 
reference pixels in a picture frame captured using the same image sensor. 
Since the dark reference pixels are captured at the same time as the rest of 
the picture frame, they may be considered to be part of the ideal dark frame 
and may therefore be used to better estimate the ideal dark frame. 

Figures 1 and 2 illustrate an embodiment of the invention as a 
method of improving the accuracy of the response of an image sensor by 
performing an adjusted dark frame subtraction. In Figure 1, two frames are 
shown, a dark frame 116 and a picture frame 126. These frames are obtained 
from the same image sensor in no particular order. Each frame contains a 
number of reference pixels 112 in the dark frame and 122 in the picture 
frame that are typically situated in the periphery of each frame. These are 
known as dark reference pixels generated by sensor elements that are not 
exposed to any incident light. 

The picture frame 126 includes a portion F p which in this 
embodiment represents the entire image of a scene captured using the 
image sensor. The sensor elements that produced F p also produce F D ' of the 
dark frame 116 in a different exposure. A pixel in F D ' contains a 
contribution from "dark electrons" only, whereas the corresponding pixel 
in F p contains contributions from dark electrons as well as photogenerated 
electrons of the same sensor element. However, due to differences in 
temperature that develop in the image sensor between the time at which 
the picture frame 126 is captured and the time at which the dark frame 116 
is captured, the contribution from dark electrons is different in the two 
corresponding pixels. 

To help compensate for such a difference, the pixel in F' D is 
adjusted in accordance with a compensation value that is related to dark 
reference pixels R } in the picture frame 126. This is represented as equation 
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(1-1) in the embodiment of Figure 1. An adjusted dark frame portion F" D is 
given by multiplying the dark frame portion F D ' by a ratio of the sum of N 
corresponding dark reference pixels R\ in the dark frame and Rj in the 
picture frame 126. The adjusted dark frame portion F" D may be viewed as a 
better estimate of the hypothetical ideal dark frame. Once the portion F' D 
has been adjusted, the adjusted dark frame F" D is subtracted from its 
corresponding picture frame portion F P to yield a corrected picture frame 
portion F P(correcled) as indicated in equation (1-2). 

Figure 2 summarizes the steps performed in one embodiment 
of the invention using the framework of Figure 1 described above. 
Operation would normally begin with step 210 in which the dark frame 116 
is captured using an image sensor. Operation continues with step 220 in 
which the picture frame 126 is captured using the same image sensor. 
Thereafter, operation continues with step 230 in which a portion of the dark 
frame is adjusted according to compensation values that are related to the 
dark reference pixels R 4 of the picture frame. This yields an adjusted dark 
frame portion F" D which is then subtracted from a corresponding portion F p 
of the picture frame to obtain a corrected picture frame portion. The 
corrected picture frame will thus receive the benefit of an improved 
estimate for the ideal dark frame. Operation may continue with step 250 in 
which further processing or display of the corrected picture frame portion 
may be performed. For instance, the corrected picture frame may be 
particularly useful in improving the accuracy of color calibration of image 
sensors. In such a scenario, the corrected picture frame will be used to 
determine a color calibration matrix for the image sensor. The effect of 
subtracting an improved estimate of the dark frame may be appreciated by 
performing a measurement capability analysis upon the color calibration 
scheme using known techniques in the art. 

The corrected picture frame portion may also be useful in 
improving the accuracy of detecting dead pixels. Reducing the error in the 
dark frame portion F' D that is subtracted from the picture frame portion F P 
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helps better identify those pixels in F P that are truly dead, i.e., have pixel 
values that are either extremely high or extremely low. 

The dark reference pixels Rj in the embodiment of Figure 1 
form at least one column of pixels in the picture frame 126. Normally, two 
or three columns are used for R,, where the selected columns are in the 
middle of a group of dark reference columns. The middle columns are 
selected to avoid the possibility of using pixels generated by exposed sensor 
circuit elements near the boundary with F P . Thus, for an image sensor with 
a maximum resolution of 1024x768, N in equation (1-1) of Figure 1 would be 
approximately several thousand. 

As one alternative to the columns of R ; , Figure 3 shows a 
group of dark reference pixels Sj which include both rows and columns that 
form a boundary around the portion F p . In this embodiment, the 
compensation values are related to a sum of dark reference pixels S s as 
indicated in equation (3-1). This results in adjusted frame portion F"' D 
which is also subtracted from its corresponding picture frame portion F P to 
give a corrected picture frame portion as indicated in equation (3-2). 

In yet another embodiment shown in Figure 4, the dark 
reference pixels are formed as a number of portions T,, T 2 , T Q . In that 
case, the dark frame 116 is divided into portions F' D1 , F' D2 , F'dq, where 
each respective portion F'^ is adjusted based on compensation values 
related to the respective reference pixels T f as indicated in equation (4-1). 
Once all of the portions F'^ have been adjusted, each is separately subtracted 
from its corresponding picture frame portion F Pj as indicated in (4-2) to 
obtain the respective corrected picture frame F Pj(corrccled) . By treating the dark 
frame 116 in this way, errors in the dark frame 116 that were caused by a 
temperature gradiant across the image sensor may be reduced. 

If segmenting the picture frame 126 as in Figure 4 were to 
result in a corrected picture frame having seams in the boundary between 
the adjacent corrected portions F^,,^, such seams may be removed by, for 
instance, applying a DC matching algorithm to the adjacent corrected 
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portions. For instance, a known DC matching algorithm such as one used 
in Joint Photographic Experts Group (JPEG) extensions may be used that 
allows the boundary between adjacent corrected portions to line up with a 
JPEG macro block. 

This disclosure so far has described techniques for reducing 
errors in the dark frame 116 that may have been caused by a temperature 
difference in the image sensor between the time at which the picture frame 
was captured and the time at which the dark frame is captured. The 
embodiment of Figure 4 in particular also addresses errors caused by a 
temperature gradient across the image sensor. Yet a third contribution to 
the error in the dark frame 116 is random noise. Random noise results in 
random pixels having random deviations in their values in both the dark 
frame and the picture frame. In contrast with the errors caused by a 
temperature difference or temperature gradient, random noise errors in 
general cannot be suppressed or corrected for using the adjusted dark frame 
subtraction techniques described above. However, in order to further 
reduce random noise in the dark frame, those techniques may be combined 
with a technique that uses a greater integration or exposure time for 
obtaining F' D as compared to F P . 

To help further alleviate random noise, the embodiments of 
Figures 1-4 above may be modified so that the integration time used to 
obtain the dark frame 116 is longer than the integration time for the picture 
frame 126. The methods described above otherwise remain identical except 
for an additional step of adjusting F' D in accordance with second 
compensation values that are related to the longer integration time 
associated with the dark frame. For instance, the adjusted dark frame 
portion F" D in equation (1-1) of Figure 1 is further adjusted as follows: 

mi if 4- 

F D =F D x-j7, f>t 
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where t is the integration time used to obtain the picture frame 126 and t' is 
the longer integration time of the corresponding pixels in the dark frame 
116. 

So far this disclosure has been referring to a single picture 
frame and a single dark frame when describing the various embodiments of 
the invention. Figure 5 shows another embodiment of the invention 
particularly applicable to a sequence of picture frames 126 M1 , 126 M2 , ... that 
may be used to capture motion in the scene. The principles that were 
discussed with the single frame embodiments above may be extended to a 
sequence of multiple picture frames. In Figure 5, a series of separate motion 
frames MF k are shown that are captured by the same image sensor. The 
frames MF k are corrected by subtracting an adjusted dark frame F" Dk as 
indicated in (5-1). In this embodiment, a single dark frame 116 D is obtained 
by the same image sensor that captures the motion frames. Each adjusted 
dark frame portion F" Dk is obtained by adjusting F' D in accordance with 
compensation values related to the dark reference pixels MR ki of the 
respective motion frame 126 Mk , as indicated in (5-2). Thereafter, the 
adjusted dark frame portions F" Dk are subtracted from the respective MF k to 
obtain the corrected motion frame portion MF k(corrected) . 

As an alternative to using a single dark frame 116, more than 
one dark frame may be captured and used with the motion frames 126 Mk . 
For instance, a separate dark frame can be captured and used to adjust every 
five motion frames. In this way, the ideal dark frame may be better 
estimated (as the temperature of the image sensor changes with successive 
frames) by keeping the dark frame and its associated motion frames closer to 
each other in time, and therefore closer in temperature. 

Figure 6 illustrates several embodiments of the invention as 
apparatuses that are part of an imaging system. In one embodiment, an 
imager module 618, such as one provided to an original equipment 
manufacturer of digital cameras, may be equipped with logic circuitry such 
as an application specific integrated circuit (ASIC) that is configured to 
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perform steps 210-240 of Figure 2 in improving the accuracy of the response 
of the image sensor 617 aboard the module. The corrected picture or 
motion frames are transferred to a processing system 610 by means of a 
removable non-volatile storage device or a computer peripheral bus (e.g., 
Universal Serial Bus). The corrected picture frames may then be further 
processed and used according to known techniques, such as for color 
calibration purposes or image processing prior to display. 

As an alternative to the application specific logic circuitry, a 
programmed processor may be used to perform the steps of adjusting the 
dark frames and performing the subtraction from the picture frames. In 
that case, the imager module 618 or digital camera 614 would include an 
article comprising a machine readable medium such as semiconductor 
memory that includes instructions which when executed by the processor 
cause the steps of Figure 2 to be performed. 

In the above described imager module and digital camera 
embodiments of the invention, the picture or motion frames were first 
corrected and then transferred to the processing system 610 being, for 
example, a computer. Alternatively, the correction may be performed by 
the image processing system 610. In that case, the system 610 is configured 
to perform the steps of Figure 2 upon receiving uncorrected dark and 
picture frames from the imager module 618 or digital camera 614. In that 
embodiment, the memory 630 and /or the mass storage device 634 (e.g., hard 
disk drive) would include instructions that when executed by the processor 
626 cause the uncorrected dark and picture frames to be transferred into the 
system 610 through the I/O control device 622 (e.g., USB controller) before 
being adjusted and subtracted as in Figures 1-5. 

The embodiments of the invention described above are, of 
course, subject to other variations in structure and implementation. For 
instance, although the adjusted dark frame subtraction techniques described 
above may be particularly useful to compensate for dark current noise in 
CMOS sensors, the techniques may be applied to alleviate the undesirable 



iOOCID: <WO 



9962023A1 I 



WO 99/62023 PCT7US99/1 0699 

10 

effects of dark currents in other types of image sensors such as infrared 
thermal imaging and charge coupled device (CCD) sensors. In general, the 
scope of the invention should be determined not by the embodiments 
illustrated but by the appended claims and their legal equivalents. 
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Claims 

What is claimed is: 

1. A method comprising: 

adjusting at least one portion of a dark frame in accordance with 
compensation values related to dark reference pixels of a picture frame to 
obtain an adjusted dark frame portion, the dark frame and the picture frame 
having been obtained from the same image sensor; and 

subtracting the adjusted dark frame portion from a corresponding 
portion of the picture frame to obtain a corrected picture frame. 

2. The method of claim 1 wherein the compensation values are related 
to a ratio of a sum of the dark reference pixels in the picture frame to a sum 
of corresponding pixels in the dark frame. 

3. The method of claim 1 wherein the dark reference pixels form at least 
one column of pixels in the picture frame. 

4. The method of claim 1 wherein the step of adjusting comprises 
adjusting a plurality of portions of the dark frame in accordance with 

compensation values related to the dark reference pixels to obtain a 

plurality of adjusted dark frame portions, 

and wherein the step of subtracting comprises 

subtracting the plurality of adjusted dark frame portions from a 

corresponding plurality of portions of the picture frame to obtain a plurality 

of corrected portions. 

5. The method of claim 4 further comprising 

removing any seams between the plurality of corrected portions. 

6. The method of claim 1 further comprising 
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adjusting the dark frame in accordance with second compensation 
values related to integration times used in obtaining the picture frame and 
the dark frame, the integration time for the dark frame being longer than 
the integration time for the picture frame. 

7. The method of claim 1 further comprising 

generating a color calibration matrix for the image sensor after 
obtaining the corrected picture frame. 

8. The method of claim 1 further comprising 
detecting dead pixels in the corrected picture frame. 

9. A method comprising: 

generating a plurality of adjusted dark frame portions based on a dark 
frame and a plurality of respective picture frames, each adjusted dark frame 
portion being obtained by adjusting a portion of the dark frame in 
accordance with compensation values related to dark reference pixels of the 
respective picture frame, the respective picture frames and the dark frame 
having been obtained using the same image sensor; and 

subtracting the plurality of adjusted dark frame portions from the 
plurality of respective picture frames to obtain a plurality of corrected 
picture frames. 

10. An imager comprising 
an image sensor; and 

logic circuitry configured to adjust at least one portion of a dark frame 
in accordance with compensation values related to dark reference pixels of a 
picture frame to obtain an adjusted dark frame portion, the dark frame and 
the picture frame having been obtained from the image sensor, and subtract 
the adjusted dark frame portion from a corresponding portion of the picture 
frame to obtain a corrected picture frame. 
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11. The imager of claim 10 further comprising 

means for transferring the corrected picture frame to an image 
processing system. 

12. The imager of claim 10 further comprising 

removable non-volatile storage device for transferring the corrected 
picture frame to an image processing system. 

13. An article comprising: 

a machine-readable medium having instructions that when executed 
by a processor cause: 

adjusting at least one portion of a dark frame in accordance with 
compensation values related to dark reference pixels of a picture frame to 
obtain an adjusted dark frame portion, the dark frame and the picture frame 
having been obtained from the same image sensor; and 

subtracting the adjusted dark frame portion from a corresponding 
portion of the picture frame to obtain a corrected picture frame. 

14. The article of claim 13 wherein the machine-readable medium has 
further instructions that when executed by the processor further cause 
generating the compensation values as being related to a ratio of a sum of 
the dark reference pixels in the picture frame to a sum of corresponding 
pixels in the dark frame. 
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